The variation of light in a scene captured by an imaging device can vary greatly. For example, objects located in a shadow of the scene can appear very dark compared to an object illuminated by direct sunlight. The limited dynamic range and colour gamut provided by traditional low dynamic range (LDR) images often do not provide a sufficient range for accurate reproduction of the changes in luminance and colour within such scenes. Typically the values of components of LDR images representing the luminance or colour of pixels of the image are represented by a limited number of bits (typically 8, 10 or 12 bits). The limited range of luminance provided by such representation does not enable small signal variations to be effectively reproduced, in particular in bright and dark ranges of luminance.
High dynamic range imaging (also referred to as HDR or HDRI) enables a greater dynamic range of luminance between light and dark areas of a scene compared to traditional LDR images. This is achieved in HDR imaging by extending the signal representation to a wider dynamic range in order to provide high signal accuracy across the entire range. In HDR images, component values of pixels are usually represented with a greater number of bits (for example from 16 bits to 64 bits) including in floating-point format (for example 32-bit or 16-bit for each component, namely float or half-float), the most popular format being openEXR half-float format (16-bit per RGB component, i.e. 48 bits per pixel) or in integers with a long representation, typically at least 16 bits. Such ranges correspond to the natural sensitivity of the human visual system. In this way HDR images more accurately represent the wide range of luminance found in real scenes thereby providing more realistic representations of the scene. High Dynamic Range imaging is becoming widely used in both the computer graphics and image processing communities.
Some display devices, however have a limited dynamic range that is inadequate for reproducing the full range of light intensities provided by HDR imaging. To this end various techniques have been used to render HDR image data compatible with LDR type displays. Tone mapping, for instance, is a technique used to map one set of colors to another in order to approximate the appearance of high dynamic range images in a medium that has a more limited dynamic range. Tone Mapping Operators (TMOs) enables the wide range of values available in a HDR image to be reproduced on a LDR display (Low Dynamic Range).
There are two main types of TMOs: global and local operators.
Global operators use characteristics of a HDR frame, to compute a monotonously increasing tone mapping curve for the whole image. As a consequence, these operators ensure the spatial brightness coherency. However, they usually fail to reproduce finer details contained in the HDR frame. Local operators tone map each pixel based on its spatial neighborhood. These techniques increase local spatial contrast, thereby providing more detailed frames.
Applying a TMO separately to each frame of an input video sequence usually results in temporal incoherency. There are two main types of temporal incoherency: flickering artifacts and temporal brightness incoherency.
Flickering artifacts are due to the TMO and are caused by rapid changes of the tone mapping in successive frames. As a consequence, similar HDR luminance values are mapped to different LDR values. Such flickering artifacts due to the TMO are undesirable and should be reduced.
Temporal brightness incoherency includes short-term and long-term temporal brightness incoherency. Short-term temporal brightness incoherency appears when rapid changes of illumination condition (global or local) occur in a HDR scene. Applying a TMO without taking into account temporally close frames results in different HDR values being mapped to similar LDR values. Consequently, the tone mapping loses information on the scene that should have been preserved.
Finally long-term temporal brightness incoherency occurs when the brightness of the relative HDR frames are not preserved during the course of the tone mapping process. Consequently, frames perceived as the brightest in the HDR sequence are not necessarily the brightest in the LDR sequence. Unlike flickering artifacts and short-term temporal brightness incoherency, long-term temporal brightness incoherency does not necessarily appear through successive frames.
In summary, applying a TMO, global or local, to each frame separately of an HDR video sequence, results in temporal incoherency.
In an attempt to address such issues various approaches have been proposed. For example, solutions, based on temporal filtering have been proposed (Boitard R., Thoreau D., Bouatouch K., Cozot R.: Temporal Coherency in Video Tone Mapping, a Survey. In HDRi2013—First International Conference and SME Workshop on HDR imaging (2013), no. 1, pp. 1-6). Depending on the TMO, either the computed tone mapping curve or the variable that adapts the mapping to the picture is filtered. Examples of such variables are the geometric mean of a picture (which is an indication of the overall brightness of a picture), its maximum or minimum value etc. However, these techniques only work for global TMOs, since local TMOs have a non-linear and spatially varying tone mapping curve. In addition, when short-term temporal brightness incoherency occurs, these techniques filter both illumination conditions together resulting in tone mapping in a transition state that corresponds to neither of the illumination conditions of the original HDR scene.
For local TMOs, preserving temporal coherency consists in preventing high variations of the tone mapping over time and space. A solution, based on the GDC operator, has been proposed by Lee et al. (Lee C., Kim C.-S.: Gradient Domain Tone Mapping of High Dynamic Range Videos. In 2007 IEEE International Conference on Image Processing (2007), no. 2, IEEE, pp. III-461-III-464.).
First, this technique performs a pixel-wise motion estimation for each pair of successive HDR frames and the resulting motion field is then used as a constraint of temporal coherency for the corresponding LDR frames. This constraint ensures that two pixels, associated through a motion vector, are tone mapped similarly.
Despite the visual improvement resulting from this technique, several shortcomings still exist. First, this solution depends on the robustness of the motion estimation. When this estimation fails (occlusion of objects), the temporal coherency constraint is applied to pixels belonging to different objects, usually resulting in ghosting artifacts. Such a motion estimation problem will be referred to as non-coherent motion vector. This issue also arises when Short-term temporal brightness incoherency occurs. In this case, the technique levels the tone mapped value to be closer to the one in the previous frame in the LDR sequence. Moreover, this technique is designed for only one local TMO, the GDC operator, and cannot be extended to other TMOs.
Finally, Guthier et al (Guthier, B., Kopf, S., Eble, M., & Effelsberg, W. (2011). Flicker reduction in tone mapped high dynamic range video. In Proc. of IS&T/SPIE Electronic Imaging (EI) on Color Imaging XVI: Displaying, Processing, Hardcopy, and Applications (p. 78660C-78660C-15)) designed a technique that reduces flickering artifacts by post-processing the output of any TMO using only information from the tone mapped sequence.
This method compares the geometric mean (which is an indication of the overall brightness of a picture) between successive frames of a video sequence. A flickering artifact is detected if this difference is greater than a threshold. As soon as an artifact is located, it is reduced using an iterative brightness adjustment until reaching the brightness threshold.
This solution detects any temporal artifacts. Consequently, brightness changes in the HDR video sequence, that are greater than the brightness threshold, are reduced during the tone mapping process resulting in short-term temporal brightness incoherency. In addition, temporal incoherencies are only considered in a global fashion and local temporal incoherencies are ignored.